Why do my event or user counts in Localytics not match my internal database or App Store Developer analytics?

In general, it is common to have some variance between different reporting systems. This is caused by how different systems define, capture, and process data (as explained in detail later). Data analysts acknowledge this and typically accept variance below a certain threshold (for example, 10%). Some suggest that "normal variance" is in the range of 15-20%.

To illustrate, suppose you see a difference in the number of sign-ups reported by Localytics versus an internal system of record. Typically, the cause is one or more of the following:

Definition of measurement

Matching data between systems can be tricky because if you are sufficiently precise, definitions often differ. This can result in what we call measurement variance.

Localytics measures unique users who download your app and have a session. Some other data sources (for example, App Store Developer analytics) count either the number downloads or the number of unique user-installs. There will usually be differences in user counts because some users will download your app but never launch it, or download then uninstall and reinstall.

With data, definitional nuances matter. In the example earlier, your app may be configured to tell Localytics about a sign-up event once the user hits the "Sign up" button in the app, while you may be comparing this data to the number of records in a local sign-up database. Speaking precisely, Localytics is measuring the number of times the sign-up button is pushed while your internal system is measuring the number of successful sign-ups. These may legitimately differ, for example, because your internal system performs additional validation before creating an actual sign-up record in the database.

Moreover, there are often implicit assumptions embedded within these definitions. One might reasonably assume that the number of occurrences of the "sign up" event should match the number of users that have signed up. But, if I can push the sign up button twice, then that assumption doesn't hold. Or, I might assume that two behaviors are necessarily tied together - for example, that by hitting the "sign up" button I must arrive on a "sign up complete" screen; however, if I can hit the signup button then hard-close the app or otherwise navigate away, this assumption doesn't hold. In short, you should be careful when you assume certain relationships exist or that certain behaviors must occur in a certain order. Often, these assumptions are sensible, but not technically valid.

Instrumentation

Even if you are careful about defining the same behavior so that it is measured in a consistent way across systems, it's possible that the event has not been instrumented as expected. This results in what can be called mechanical variance, meaning it arises from how your app has been mechanically configured to collect data.

In Localytics, the number of occurrences of an Event is determined by how the Event has been configured in your app. When users interact with your app, Event tags are like digital tripwires that can be activated when an action occurs. If a user performs an action which triggers that tripwire, Localytics creates a timestamped event. But, if the tripwire is located in the wrong place or inadvertently captures unintended behaviors, the resulting data will be technically legitimate but different than anticipated. For example, it might be possible that the sign-up event legitimately fires only under certain conditions, not all conditions, meaning that not all code paths have been considered by your Developers.

In short, your developers control the logic of how and when to pass information to Localytics, and sometimes when to upload that information to Localytics. If Event or User counts seem incorrect, we recommend performing Quality Assurance testing to validate that your Event "tripwires" are capturing the intended behavior exactly once.

Processing

Even if you've been careful about definition and instrumentation, it's still possible you observe what we call processing variance between systems, which results from differences in data ingestion pipelines.

For example, Localytics has logic to capture behavior even if that behavior occurs offline (such data is subsequently sent to Localytics at the next available opportunity). Other systems might not be set up to handle offline data, or may apply a different timestamp to the data based on when it is received by your servers versus when the behavior actually occurred. Localytics also has a sophisticated data ingestion pipeline with advanced error-handling, de-duplication, and other logic. There may be scenarios, for example, where data tries to upload multiple times due to network conditions, and Localytics is smart enough to de-duplicate events like this. Other systems may not have this logic or may have different logic.

So, what can I do to reduce variance across systems?

  • Review tagging code

    Typically, when there is variance between different systems, the best approach is to review your app and code to identify which of the three reasons earlier is most likely. We most often find that it is either #1 or #2, or both. Sometimes developers will notice meaningful differences after a detailed code review or will realize that they haven't considered all possible behavioral paths when instrumenting the app.

  • QA event-tagging:

    We recommend QA'ing the event-tagging to ensure that events are firing in the intended manner. Using a tool like Charles Proxy, you can "watch" the events that are passing to Localytics after performing behaviors in the app. This allows you to validate and improve tagging in real-time.

  • QA SDK integration:

    Your developers should confirm that the Localytics SDK has been integrated as recommended.